Fault tolerant virtual private network endpoint node

ABSTRACT

A provider network includes a service that creates fault tolerant virtual private network (VPN) endpoint nodes. Each such VPN endpoint node is created as a plurality of virtual machines executing on host computers. Each of the virtual machines is configured from a common machine image that includes software capable of causing the respective virtual machine to configure a secure communication tunnel such as an IPSec tunnel. One of the virtual machines, however, is operated in an active mode to actively configure the tunnel and send and receive encrypted traffic over the tunnel, while another virtual machine is configured to operate in a standby mode. The standby mode VPN endpoint virtual machine can be quickly transitioned to the active mode to take over the role of configuring and exchanging encrypted packets over the tunnel should the active mode VPN endpoint experience a failure.

CROSS-REFERENCE TO RELATED APPLICATION

This disclosure contains subject matter that may be related to subjectmatter in application titled “Peered Virtual Private Network EndpointNodes” having U.S. application Ser. No. 15/277,962 (U.S. Pat. No.10,397,189), filed on the same day as the present application andincorporated by reference.

BACKGROUND

Many companies and other organizations operate computer networks thatinterconnect numerous computing systems to support their operations,such as with the computing systems being co-located (e.g., as part of alocal network) or instead located in multiple distinct geographicallocations (e.g., connected via one or more private or publicintermediate networks). For example, data centers housing significantnumbers of interconnected computing systems have become commonplace,such as private data centers that are operated by and on behalf of asingle organization, and public data centers that are operated byentities as businesses to provide computing resources to customers. Somepublic data center operators provide network access, power, and secureinstallation facilities for hardware owned by various customers, whileother public data center operators provide “full service” facilitiesthat also include hardware resources made available for use by theircustomers.

The advent of virtualization technologies for commodity hardware hasprovided benefits with respect to managing large-scale computingresources for many customers with diverse needs, allowing variouscomputing resources to be efficiently and securely shared by multiplecustomers. For example, virtualization technologies may allow a singlephysical computing machine (e.g., a server) to be shared among multipleusers by providing each user with one or more virtual machines hosted bythe single physical computing machine, with each such virtual machinebeing a software simulation acting as a distinct logical computingsystem that provides users with the illusion that they are the soleoperators and administrators of a given hardware computing resource,while also providing application isolation and security among thevarious virtual machines. Communication pipelines can be establishedfacilitating traffic between virtual machines and between logicalgroupings of virtual machines.

BRIEF DESCRIPTION OF THE DRAWINGS

For a detailed description of various examples, reference will now bemade to the accompanying drawings in which:

FIG. 1 shows an example of a virtual private network (VPN) attached to afault tolerant VPN endpoint node coupled by a secure tunnel to a remotegateway in accordance with various embodiments;

FIG. 2 shows an implementation of a provider network in accordance withvarious examples;

FIG. 3 shows a method in accordance with various examples;

FIG. 4 shows an example of an implementation of a fault tolerant VPNendpoint node that submits heartbeat messages for analysis by a healthmonitoring service in accordance with various embodiments;

FIGS. 5-9 illustrate the detection of a failure of an active mode VPNendpoint virtual machine and fail-over to a standby VPN endpoint virtualmachine in accordance with various embodiments;

FIG. 10 shows a method for detection of a failed active mode VPNendpoint virtual machine, fail-over to the standby mode VPN endpointvirtual machine, and replacement of the failed active mode VPN endpointvirtual machine with a new virtual machine;

FIG. 11 shows an embodiment of a synchronization process forsynchronizing a newly computed key between the active and standby modeVPN endpoint virtual machines;

FIG. 12 shows another embodiment of a key synchronization process;

FIG. 13A shows an example of the use of the fault tolerant VPN endpointnode to which create a tunnel between a pair of virtual private networksin accordance with various embodiments;

FIG. 13B illustrates a method of peering two VPN endpoint nodes toestablish a tunnel between the peered VPN endpoint nodes and theirrespective virtual private networks;

FIG. 14 shows another example of multiple virtual private networksinter-connected by tunnels formed between fault tolerant VPN endpointnodes attached to each such virtual private network in accordance withvarious embodiments;

FIG. 15 illustrates multiple virtual private networks of a customercoupled by respective secure tunnels implemented by fault tolerant VPNendpoint nodes to a gateway in the customer's datacenter in accordancewith various embodiments;

FIG. 16 illustrates the use of a dedicated router in a collationfacility to which a secure tunnel can be created by use of a faulttolerant VPN endpoint node in accordance with various embodiments; and

FIG. 17 shows a block diagram of computing node usable to implement anyof the hardware and software components described herein in accordancewith various embodiments.

DETAILED DESCRIPTION

A provider network is described herein that permits customers to requestthe creation of a fault tolerant virtual private network (VPN) endpoint(VPNe) node and to then connect the fault tolerant VPN endpoint to avirtual private network which includes one or more of the customers'virtual machines. A customer can then cause a secure tunnel to beestablished between the VPN endpoint (and thus the customer's virtualprivate network) and a remote node such as another VPN endpoint node, anetworking device such as a gateway on the customer's premise, etc.Another network may be attached to the remote node. In response to arequest submitted by the customer for creation of the fault tolerant VPNendpoint node, a provisioning service within the provider network causesa pair of virtual machines to be launched from a machine image thatcontains an application that implements VPN endpoint functionality. Bothvirtual machines contain an application that performs the functionalityof a VPN endpoint, including implementing one or protocols forestablishing a secure tunnel to a remote node, recalculating keys suchas encryption keys, negotiating security protocols with the remote node,etc. In one embodiment, the protocol implemented by the VPN endpointvirtual machine application includes the Internet Protocol Security(IPSec) protocol.

One of the VPN endpoint virtual machines that is created to implementthe fault tolerant VPN endpoint node can be assigned a public IP addresswhich it uses, as well as other information, to establish the tunnelwith the remote node, and the other VPN endpoint virtual machine is notassigned the public IP address and thus is not able to establish thetunnel. The VPN endpoint virtual machine that is presently assigned thepublic IP address and thus can establish the tunnel is referred toherein as the “active mode” VPN virtual machine, and the VPN endpointvirtual machine that presently is not assigned the public IP address isreferred to as the “standby mode” VPN virtual machine. The active andstandby modes can be switched between the VPN virtual machines such as,for example, during a failover process as described herein. The faulttolerant VPN endpoint node can be created with more than one standbymode VPN endpoint virtual machine to provide additional degrees of faulttolerance. The active mode VPN endpoint virtual machine is configured bythe provisioning service with an elastic IP address that is used, alongwith other state information configured into the virtual machine, toestablish the secure tunnel to the remote node over which encryptedpackets between the customer's private network and a network attached tothe remote node can be exchanged. The standby mode VPN endpoint virtualmachine also has an application that can permit that virtual machine tocontinue the operation and control of the tunnel to the same remote nodeshould the active mode VPN endpoint virtual machine be unable tocontinue satisfactory operation. While in the standby mode, the standbymode VPN endpoint virtual machine, however, does not have the IP addressfor the tunnel and thus does not actually establish and control thetunnel with respect to the remote mode. The standby mode VPN endpointvirtual machine, however, can be transitioned to the active mode (e.g.,by moving the IP address from the active mode VPN endpoint virtualmachine to the standby mode VPN endpoint virtual machine) if the activemode VPN endpoint virtual machine experiences a failure. In someembodiments, the provider network implements a health monitoring servicethat monitors heartbeat messages from the active and standby mode VPNendpoint virtual machines. Each heartbeat message includes informationabout the health of the respective virtual machine. If the active modeVPN endpoint virtual machine reports a failure or insufficient heartbeatmessages are received, the health monitoring service may initiate afail-over from the active mode VPN endpoint virtual machine to thestandby mode VPN endpoint virtual machine. That is, the standby mode VPNendpoint virtual machine is transitioned to the active mode and thetunnel is connected to that virtual machine instead of the failed VPNendpoint virtual machine. The local (i.e., public) IP address of the VPNendpoint is re-assigned from the failed VPN endpoint virtual machine tothe former standby mode VPN endpoint virtual machine, which is now inthe active mode. The secure tunnel continues to operate albeit from thenewly activated VPN endpoint virtual machine thereby providing faulttolerance.

In some embodiments, the VPN endpoint virtual machines may implement atunnel protocol that implements keys such as encryption keys and thatperiodically specifies that the keys should be recomputed. Foe example,the IPSec protocol includes a Phase I Diffie-Hellman key and a Phase IIIPsec encryption key. Both keys are periodically recomputed, albeit atseparately configurable rates. In such embodiments, the provider networkmay include a key store service that permits the active mode VPNendpoint virtual machine to synchronize any new key it computes with thestandby mode VPN endpoint virtual machine so that the standby mode VPNendpoint virtual machine can quickly take over the role of the activemode VPN endpoint virtual machine and can thus use the most recentlycomputed key to operate the tunnel (e.g., to encrypt packets transmittedacross the tunnel).

FIG. 1 illustrates a provider network 90 including one or more virtualprivate networks 100. Each customer of the service provider can requestservices executing with the provider network to launch one or morevirtual machines 110 for use by the customer, which the customer canthen use for any customer-desired purpose (e.g., hosting a website,batch processing, etc.). The request to launch virtual machines may bein the form of requests to application programming interfaces (APIs). Agroup of the customer's virtual machines may be configured to form avirtual private network 100. Each virtual machine comprises avirtualization of a physical computing device such as a server andcomprises code such as an operating system, drivers, and applicationsthat execute on a host computer.

In accordance with the disclosed embodiments, the customer can submit anAPI request to the provider network to create a fault tolerant VPNendpoint node 120, which the customer then can request to be attached tothe customer's virtual private network 100. The customer also may use adata center 150 separate from the service provider's provider network90. The customer can configure a remote node within his premise's datacenter 150 such as a gateway 152. The gateway 152 may couple to one ormore servers 154 or other types of computing devices forming one or morenetworks within the customer' data center 150. A secure (e.g.,encrypted) tunnel 123 can be established between the fault VPNe endpointnode 120 within the provider network and the remote gateway 152 overwhich encrypted traffic can be transmitted between the virtual privatenetwork 100 within the provider network and a network within thecustomer's own data center 150.

The requests to create the virtual machines 110, form the virtualprivate networks of virtual machines, and create the fault tolerant VPNendpoint 120 may be in the form of, for example, a request to an APIthat may be processed by a provisioning service executing within theprovider network 90. The API request to create the VPN endpoint node 120may be to a CreateVpnEndpoint API and may include various inputparameters such as any one or more of a remote IP address, a remotepre-shared key (PSK), a tunnel inside IP classless inter-domain routing(CIDR) block of addresses, a remote Border Gateway Protocol (BGP)Autonomous System (AS) number (in embodiments in which BGP isimplemented), and a local BGP AS number. The remote IP address is thepublic IP address of the remote node to which the customer's faulttolerant VPN endpoint node 120 is to form a tunnel over which encryptedtraffic will flow between the customer's virtual private network 100 andthe network(s) in the data center 150 that are connected to gateway 152.The remote pre-shared key is a key that is preconfigured into the VPNendpoint node 120 and the gateway and is used as part of a tunnelingprotocol such as the IPSec protocol to authenticate each end node of thetunnel to the its peer attempting to form an IPSec tunnel. The tunnelinside IP CIDR block may comprise a plurality of IP addresses used toestablish connectivity of the fault tolerant endpoint node within theprovider network. One of the IP addresses in the CIDR block is the IPaddress of the VPN endpoint node and another IP address is the IPaddress of the remote node. The remote and local BGP AS numbersdesignate the particular autonomous systems that the fault tolerant VPNenode 120 and its remote peer counterpart should use when establishingthe tunnel therebetween. The input parameters also may specify whether afault tolerant or non-fault tolerant VPN endpoint node is to be created.

In response to receiving the CreateVpnEndpoint API request for thecreation of a fault tolerant VPN endpoint node, a provisioning servicewithin the provider network (shown in FIG. 2 and discussed below)selects multiple host computers operating within the provider networkand causes a virtual machine to be launched on each such host. In oneembodiment, the provisioning service selects two separate host computerson which to launch the virtual machines, although one host computercould be used to launch the virtual machines. The virtual machines arelaunched by causing a particular machine image to be retrieved frommachine image storage and transmitted to the selected host computer. Themachine image may include an operating system, drivers, and a VPNendpoint application. The VPN endpoint application causes the virtualmachine to perform the various operations needed to implement a VPNendpoint. Such operations depend on the particular protocols used toimplement the VPN tunnel. In the case of using IPSec to implement thetunnel, the operations performed by the VPN endpoint application mayinclude authenticating the remote node based on the pre-shared key,exchanging certificates, generating a Diffie-Hellman key, generating anIPSec key, negotiating security protocols and key lifetimes with theremote node, encrypting packets to be sent over the tunnel, decryptingpackets received over the tunnel, renegotiating a security association(including keys), etc. The VPN endpoint applications may implement apair of opposite direction communication paths (send and receive) toform the tunnel. If a non-fault tolerant VPN endpoint node is to becreated, a single virtual machine is launched to implement the VPNendpoint node.

The machine images may be stored in a centralized database within theprovider network. Each machine image, including the VPN endpoint-basedimages as well as other types of machine images usable by customers tolaunch other types of virtual machines may have pre-assigned identifiers(IDs). The IDs may be used by the provisioning service to launch avirtual machine as a VPN endpoint node. The provisioning service selectsthe machine image to copy to the host computer for launching the VPNendpoint virtual machine using the ID associated with the machine imageneeded to implement the VPN endpoint node functionality.

After the VPN endpoint-based machine images are loaded on the hostcomputers, the provisioning service also may cause configurationparameters to be downloaded into the host computer for use by therespective virtual machine. Some of the configuration parameters mayoriginate from the customer's initial API request for creating the VPNendpoint node, while other configuration parameters be stored in acentralized database or otherwise generated by the provisioning serviceor other services within the provider network. The configurationparameters may include those mentioned above such as the remote IPaddress, the remote pre-shared key (PSK), the tunnel inside IP CIDR, theremote BGP AS Number, and the local BGP AS Number. The collection of theconfiguration parameters loaded into the VPNe-based virtual machine toconfigure the virtual machine to be able to implement the secure tunnelto the remote node is referred to as “state information.”

The customer also configures the gateway 152 with parameters it needs tohelp establish the secure tunnel to the VPN endpoint node 120. Forexample, the customer may configure the gateway with the public IPaddress assigned to the VPN endpoint node 120, the type of securityprotocol(s) supported by the VPN endpoint node (e.g., AuthenticationHeaders (AH), Encapsulating Security Payloads (ESP), etc.), type ofencryption algorithm to be used, the pre-shared key, etc.

As noted above, the provisioning service causes the VPN endpoint node tobe created by launching two (or more) VPN endpoint virtual machines onone or more host computers using a suitable machine image (e.g., amachine image containing an application that implements the appropriatetunneling protocols). FIG. 1 illustrates that VPN endpoint node 120 isimplemented a VPN endpoint virtual machines 122 and 125. In variousembodiments, the two VPN endpoint virtual machines generally may beidentical. For example, the machine images both may be booted fromidentical copies of a machine image. Each VPN endpoint virtual machineis selectively configurable to operate in an active mode or in a standbymode. That is, each VPN endpoint virtual machine comprising the VPNendpoint node is capable of operating in both the active and standbymodes, albeit not at the same time. While one virtual machine is in theactive mode, the other virtual machine is in the standby mode, and viceversa. The VPN endpoint virtual machine in the standby mode providesredundancy in the event that the active mode VPN endpoint virtualmachine experiences a failure precluding it from operating correctly. Assuch, the VPN endpoint node 120 described herein is fault tolerant andthe customer need not be involved in the implementation of the faulttolerant aspect of the VPN endpoint node 120. The customer simplyrequests a VPN endpoint node to be created, and the provisioning servicewithin the provider network responds by creating two virtual machinesthat can implement the functionality of the VPN endpoint node with onevirtual machine being a standby instance to the other. In the event theactive VPN endpoint virtual machine fails (e.g., a software failure ofthe instance or a hardware failure of the server on which it executes),a failover mechanism automatically causes the standby mode VPN endpointvirtual machine to take over the role of the active instance andcontinue the operation of the secure tunnel to the remote node.

As noted above, the customer's virtual private network comprises one ormore virtual machines created by or for the customer's use and executeon servers. The fault tolerant VPN endpoint 120 also is implemented asmultiple (e.g., two) virtual machines executing on severs. A virtualmachine is a software implementation of a physical computer system.Virtual machines may provide for multiple and/or different operatingsystem environments to run concurrently on a single host computer 120.In one example, multiple virtual machines of a Linux® operating systemenvironment may execute concurrently with multiple instances of aMicrosoft® Windows® operating system environment on a single physicalcomputer. A virtual machine may interact with a hypervisor or a virtualmachine monitor (or other type of virtualization system) which areprograms that execute on the physical computer that allow multiple guestoperating systems to share a single hardware host. Each operating systemappears to have exclusive access to the host's processor, memory, andother resources. However, the hypervisor controls the host processor andresources, allocating resources that are needed to each instance'soperating system in turn and making sure that the guest operatingsystems of the virtual machines do not disrupt each other. Each virtualmachine may be controlled by a respective customer.

The virtual machines created by the customer to form the overlay networkcan be loaded by the customer with whatever customer-specificapplications the customer so chooses. For example, the customer'sapplications may comprise web server applications, data processingapplications, or any other type of functionality that the customerdesires. The applications executing within the VPN endpoint virtualmachines may be pre-stored in the machine images used to boot suchvirtual machines or may be loaded into the virtual machines post-boot bythe provisioning service.

A VPN endpoint 120 created by the provisioning service may be attached,at the request of the customer (e.g., via a request submitted to anAPI), to the customer's virtual network, although the VPN endpoints 120can be attached to other types of networks as well (e.g., a group ofvirtual machines that do not form a virtual private network. Eachvirtual network for a customer may be implemented over one or moreintermediate physical networks that interconnect computing nodes onwhich the customer's virtual machines execute. That is, a virtualnetwork may be implemented over a physical network. Each customer mayhave their instances in a virtual network. A virtual network usesvirtual IP addresses and corresponding physical IP addresses. Theimplementation of a virtual network may include modifying or addingadditional headers to packets to map virtual addresses consistent withthe virtual network to physical addresses associated with the underlyingphysical network so that the packets can be routed through the physicalnetwork between host computers. A mapping service may be provided tostore, update and provide virtual-to-physical address mappings for usein modifying packets to be transmitted between virtual machines in avirtual private network.

The virtual network may be implemented in various ways in variousembodiments, such as by using IPv4 (“Internet Protocol version 4”) orIPv6 (“Internet Protocol version 6”) packets as the overlay networkpacket size. For example, virtual network address information for avirtual network could be embedded in a larger physical packet networkaddress space used for a networking protocol of the one or moreintermediate physical networks. As one illustrative example, a virtualnetwork may be implemented using 32-bit IPv4 network addresses, andthose 32-bit virtual network addresses may be embedded as part of128-bit IPv6 network addresses used by the one or more intermediatephysical networks, such as by re-headering communication packets orother data transmissions, or otherwise modifying such data transmissionsto translate them from a first networking protocol for which they areconfigured to a distinct second networking protocol. In otherembodiments IPv4 packets could be used by the physical network and thevirtual network. For example, the size of the IPv4 packet generated bythe virtual machine could be limited to a size that can be inserted intoan IPv4 packet and leave enough bits so the service provider can addheaders to the packet.

Referring still to FIG. 1, both the active mode and the standby mode VPNendpoint virtual machines 122 and 125 have access to a shared keystorage 128. The shared key storage 128 is used by the VPN endpointvirtual machines 122 and 125 to exchange the keys used to implement thetunnel 123. In the event that the active mode VPN endpoint virtualmachine 122 fails, the standby VPN endpoint virtual machine 125, whichalready has the relevant key(s), can quickly take over role as theactive VPN endpoint virtual machine. The key exchange process isdescribed below in greater detail.

FIG. 2 shows another embodiment of the provider network 90. In thisembodiment, the provider network includes multiple host computers suchas host computers 180, 200, and 230 (which may implemented a servers)coupled via a network 175. The network 175 comprises multiple routersand switches that provide connectivity between the various computing andstorage devices within the provider network. Host computers 180 can beused to launch virtual machines 182 for customers to use as thecustomers see fit as noted above. Customers can interact via their ownuser devices 290 with a provisioning service 250 to request the creationof one or more virtual machines 182. A user device 290 may comprise anytype of computing device such as personal computer, a smart phone, atablet device, etc. Through the user device 290, a customer can submitrequests to the provisioning service for various services such asvirtual machine creation, VPN endpoint node creation, virtual machinetermination, virtual network creation, etc. The customer may submit APIrequests via a command line interface (CLI) or a script file, or both.Alternatively, or additionally, a web browser may execute on the userdevice 290 and the customer may interact with the provisioning service250 via the web browser. The provisioning service 250 may be implementeda number of constituent services. One such service may be a userinterface which generates hypertext markup language (HTML) web pageswhich are transmitted across a public network (not shown) such as theInternet to the user device for display thereon. The web browser maydisplay a console interface by which the customer can interact. Theprovisioning service may receive and process API requests from thecustomers of the service provider and perform the operations indicatedby the API requests. For example, the provisioning service may perform aworkflow to launch virtual machines on host computers, launch the faulttolerant VPNe virtual machines described herein, etc.

FIG. 2 shows an example of host computers 200 being used to executevirtual machines 210, as well as a fault tolerant VPN endpoint node 235implemented on host computers 230. The VPN endpoint node 235 isimplemented as VPN endpoint virtual machines 232 a and 232 b executed onthe host computers 230. The customer in this example also has requested,for example via additional API requests, that his virtual machines 210be formed into a virtual private network 215 (or the formation of thevirtual private network 215 may have been part of the launch request tolaunch the virtual machines 210 in the first place) and that the VPNendpoint node be connected to the customer's virtual private network215. The provisioning service 250 can launch virtual machines 182, 210and 232 a and 232 b on the various host computers as shown.

Each of the VPN endpoint virtual machines 232 a, 232 is capable of beingconfigured to be in an active mode or in a standby mode. The applicationexecuting with the virtual machines 232 a, 232 b may include aconfiguration mode setting that can be set by an external agent such asthe provisioning service. In some examples, the provisioning service 250may send a signal such as packet to a VPN endpoint virtual machine 232a, 232 b to configure it for active mode operation or standby modeoperation. The provisioning service 250 configures one of the virtualmachines for the active mode of operation and the other virtual machinefor the standby mode of operation. Thus, one of VPN endpoint virtualmachines 232 a, 232 b is an active mode VPN endpoint virtual machine andother is a standby mode VPN endpoint virtual machine, and theprovisioning service can reconfigure the standby mode VPN endpointvirtual machine to the active mode upon detection of a failure of theactive mode VPN endpoint virtual machine.

The host computers within the provider network can exchange packetsacross network 175 using IP addresses of the host computers (“physical”IP addresses). A virtual private network, however, as explained above isone in which the member virtual machines use a different set ofaddresses (“virtual” IP addresses). In accordance with some embodiments,one or more of the computers may include a virtual machine communicationmanager (VMCM) usable for the implementation of the virtual privatenetworks. Host computers 200 and 230, for example, include VMCMs 220 and230. The VMCMs 220 and 230 may modify (as described above) an outgoingpacket destined for a virtual IP address of another virtual machinewithin the customer's virtual private network based on the physical IPaddresses used within provider network. For example, if a communicationpacket is to be sent between computing nodes in the service provider'snetwork, the originating packet may include an IP address in accordancewith a particular protocol (e.g., IPv4), and a VMCM associated with thesending host computer embeds the virtual network packet into a substratenetwork packet which includes physical source and destination IPaddresses. The VMCM then transmits the packet through theinterconnection network 175. A VMCM associated with the receiving hostcomputer receives the substrate packet, extracts the virtual networkpacket and forwards the virtual network packet on to the targetedvirtual machine. A mapping service 270 is shown in FIG. 2 and may storemappings between virtual and physical IP addresses. Such mappings may betransmitted when needed to a particular VMCM 220, 236 to send andreceive packets within a given virtual private network.

The provider network 90 in FIG. 2 also includes a health monitoringservice 260 and a health monitoring database 262. The health monitoringservice may comprise machine instructions that execute on a servercomputer and the health monitoring database may be stored in a storagedevice such as hard drive, solid state storage, etc. The healthmonitoring service and database can be used to monitor the health andstatus of the VPN endpoint virtual machines 232 a, 232 b and the hostcomputers on which they execute, as well other virtual machines andcomputing devices within the provider network. In some embodiments, eachVPN endpoint virtual machine sends a heartbeat message at periodic ornear-periodic intervals. Each heartbeat message may encode health and/orstatus information about the corresponding virtual machine. The factthat a heartbeat message was sent at all indicates something about theoperational nature of the virtual machine. Failure to transmit aheartbeat message may be indicative of a failure of the virtual machineor other components within the host computer on which the virtualmachine executes. The health and status information contained within theheartbeat messages may include any of a variety of information such aserror codes indicative of any errors detected internal to the virtualmachine such as memory errors, network port timeouts, etc., processorutilization rates, memory utilization rates, etc. The health and statusinformation may contain no information about any problems and thusinclude values or metadata indicative of a healthy and fully operationalvirtual machine. In some examples, the health and status information mayinclude a healthy/unhealthy indicator for each of multiple subsystemswithin the virtual machine and corresponding virtualization system.

Each of the VPN endpoint virtual machines 232 a, 232 b sends theheartbeat messages to the health monitoring database 262 for storagetherein. The health monitoring database 262 thus may store health andstatus messages and information for multiple VPN endpoint virtualmachines. Each such virtual machine has an ID and the heartbeat messagesmay include the ID of the respective virtual machine. The healthmonitoring service 260 can access the health monitoring database 262 anddetermine the health and status of a given VPN endpoint virtual machine.As such, the health monitoring service 260 can determine whether anactive mode VPN endpoint virtual machine has failed and, as explainedbelow, if the active mode VPN endpoint virtual machine is determined tobe experiencing a failure, initiate a fail-over process to the standbyVPN endpoint virtual machine.

Referring still to FIG. 2, the provider network 90 also includes a keystorage service 280 which contains or otherwise has access to a keystore 282. The key storage service 280 is accessible the VPN endpointvirtual machines 232 a, 232 during a rekeying operation that may be partof the protocol that implements the tunnel to the remote node (e.g.,tunnel 123 in FIG. 1). The key store 282 comprises storage that can beshared exclusively by an active/standby pair of VPN endpoint virtualmachines. The keys stored in the key storage may be encrypted. The useof the key storage service 280 by a given active/standby pair of VPNendpoint virtual machines is described in detail below.

FIG. 3 illustrates a method in accordance with various embodiments. Theoperations may be performed in the order shown, or in a different order.Further, two or more of the operations may be performed concurrentlyinstead of sequentially. At 300, the method includes creating a virtualprivate network. This operation may be performed in response to receiptfrom a customer of API requests to create multiple virtual machines andthen implement a virtual network from a given set of the customer'svirtual machines. The provisioning service 250 may implement thisoperation by causing machine images to be loaded onto host computers,cause virtual machines to be booted based on the machine images,configure the virtual machines with IP addresses, etc. The customer canspecify, via an API request, which virtual IP addresses are to be usedfor the individual virtual machines within the virtual private networkand the corresponding mappings between virtual and physical IP addressesmay be added to a database within the mapping service 270.

At 302, the method may include receiving an API request to launch a VPNendpoint node. The API request may be initiated by a customer from auser device 290 as described above. The API request may be to theCreateVpnEndpoint API as noted above and include the various inputparameters explained previously (IP address of remote node, pre-sharedkey, etc.). The API request may be received and processed by theprovisioning service 250. In response to receipt of theCreateVpnEndpoint API request, the provisioning service at 304 causes apair (or more than two) of virtual machines to be launched from amachine image containing a VPN endpoint application. A repository (notspecifically shown) containing machine images may be accessed by theprovisioning service. The provisioning service may be configured to usea particular machine image (with a particular ID) when attempting tolaunch a VPN endpoint virtual machine. The provisioning service 250 mayselect a host computer and send a packet to that computer with the ID ofthe targeted machine image. The host computer then may obtain themachine image itself form the centralized storage. In other embodiments,the provisioning service may send a message to the repository containingthe targeted machine image and request a copy to be transmitted to aspecific host computer. Regardless of the mechanism implemented to placethe desired machine image on the selected host computer, the machineimage containing the VPN endpoint application is stored in a storagedevice (magnetic hard drive, solid state storage, etc.) of the hostcomputer.

At 306, the method includes transmitting to, and loading configurationparameters within, each VPN endpoint virtual machine. Examples of theconfiguration parameters are provided above and include the IP addressof the remote node to which the tunnel is to be established, the remotepre-shared key, the BPG AS numbers, etc. The provisioning service, oranother service within the provider network, may transmit theconfiguration parameters to each VPN endpoint virtual machine.

Once the VPN endpoint virtual machines are launched and operational, anID may be generated by the provisioning service 250 for the VPN endpointnode (i.e., the pair of VPN endpoint virtual machines). As the customerneed not be aware of the existence of the fact that two VPN endpointvirtual machines have been created in response to the customer'srequest, the ID that is generated and assigned is applicable to thecollective pair of virtual machines. Thus, the customer need only referto the VPN endpoint node by the one VPN endpoint ID.

At 308, the method includes configuring one of the virtual machineslaunched at 304 to be in the active mode (e.g., through assignment tothat virtual machine of a public IP address) and the other virtualmachine to be in the standby mode (e.g., through no assignment of apublic IP address). Selection of the virtual machine to be in the activemode may comprise a random selection by the provisioning service betweenthe two virtual machines, selection by the provisioning service of thevirtual machine that first acknowledges that it has completed the bootprocess back to the provisioning service, or any other methodology forselecting one of the two virtual machines. Configuring each virtualmachine to be in either the active or standby mode may comprisetransmission by, for example the provisioning service, of a message thatindicates whether the receiving virtual machine is to be in the activemode or in the standby mode. Each VPN endpoint virtual machine maycontain a storage element (memory, register, etc.). The message from theprovisioning service that indicates whether each virtual machine is tobe in the active or standby mode contains a mode indicator (activeversus standby) which may be stored in the storage element, and thenread by the application software within the virtual machine to set itsmode.

In other embodiments, the two virtual machines may negotiate betweenthemselves as to which one of them is to be in the active mode, and thenonce they decide amongst themselves which virtual machine is to be theactive mode virtual machine and which is to be the standby mode virtualmachine, transmit messages to the provisioning service 250 as to theirdetermined operational modes (i.e., messages that indicate to theprovisioning service which VPN endpoint virtual machine is in the activemode and which is in the standby mode). The inter-VPN endpoint virtualmachine negotiation may include each of the virtual machines generatinga value such as a random number and transmitting a packet with its valueto the other virtual machine. The VPN endpoint virtual machine havingthe higher value (or lower value) is the VPN endpoint virtual machinethat is to transition to the active mode, while the other VPN endpointvirtual machine transitions to the standby mode. The virtual machinesmay inform the provisioning service as to the results of the negotiationand the provisioning service may confirm the results by transmittingback to each VPN endpoint virtual machine a packet containing itscorresponding mode value (active versus standby).

At 310, the method includes the active and standby mode virtual machinesregistering with the key storage service 280. The registration processmay include each virtual machine submitting an API call to the keystorage service that the virtual machine (e.g., standby virtual machine)is to receive notification of a change in a key of the other virtualmachine (e.g., the active virtual machine). As a result, when the activevirtual machine changes a key it uses for the tunnel to the remote node,it publishes its newly computed key to the key storage service and thekey storage service informs the standby virtual machine of the keychange. In other embodiments, the provisioning service may transmit oneor more messages to the key storage service 280 with the correspondingidentifiers. The identifiers may comprise, for example, IP addresses ofthe corresponding virtual machines, and an explicit or implicit mappingbetween the identifiers which indicates that the VPN endpoint virtualmachines associated with those identifiers are counterparts of anactive/standby pair of VPN endpoint virtual machines which collectivelyform a single VPN endpoint node. In some embodiments, both identifiersmay be provided in a single message to the key storage service whichimplicitly maps together the two VPN endpoint virtual machines. In otherembodiments, the message sent to the key storage service 280 may containan explicit indicator that the virtual machines are related as activeand standby counterparts.

At 312, a customer may request that the VPN endpoint node (whichcomprises an active mode VPN endpoint virtual machine and a standby modeVPN endpoint virtual machine) is to be attached to the virtual privatenetwork created at 300. The customer may submit a request to an API (forexample called an AttachVpnEndpoint API to request the attachment. TheAPI request may be initiated via the customer's user device 290 and mayinclude, as an input parameter, the VPN endpoint ID noted above, as wellas other values as desired such as the ID of the customer's virtualprivate network created at 300, a subnet identifier value, etc. Inresponse to this API request, the provisioning service 250 may updaterouting tables within the customer's virtual private network to indicatewhich packets are to be sent to the newly created VPN endpoint node fortransmission across the tunnel.

At 314, the remote node to which the VPN endpoint node is to becommunicatively coupled via a tunnel is configured. The remote node maybe a computing device (e.g., a server configured as a gateway such asgateway 152 in FIG. 1) owned and/or operated by the customer and outsidethe control of the provider network. In such instances, the customerconfigures his remote node based on the configuration of his VPNendpoint node. In some embodiments, the customer can submit aDescribeVpnEndpoint API request which may be processed by theprovisioning or other service executed within the provider network toreturn configuration data to the customer pertaining to the VPN endpointnode. Such configuration data may include the IP address assigned by theprovider network to the VPN endpoint node, the remote node's IP address,the local BGP AS number, the remote BGP AS number, the tunnel IP CIDRblock address, etc. The customer configures his remote node to implementthe same security protocols as the VPN endpoint node. Such securityprotocols may be published or otherwise specified to the customer by theservice provider. In some implementations, the customer may have somecontrol over which security protocols are implemented by his VPNendpoint node and such security protocols may be included in theCreateVpnEndpoint API request at operation 302. The customer also mayspecify the IP address associated with the VPN endpoint node implementedin the provider network. The IP address may be returned in response tothe DescribeVpnEndpoint API request.

In some cases, the customer may want to establish a tunnel between apair of virtual private networks within the provider network 90. Thevirtual private networks are communicatively coupled by way of a VPNendpoint node attached to each virtual private network. One virtualprivate network and its VPN endpoint node are created via operations300-308. A second virtual private network may be created as describedabove in operation 300. A VPN endpoint node may be created for thesecond virtual private network in response to, for example, a request toa CreateVpnEndpointPeer API. The input parameter to this request mayinclude the VPN endpoint node ID assigned to the VPN endpoint nodecreated in operations 302-308. This API request causes the provisioningservice 250 to create a VPN endpoint node implemented as a pair ofactive/standby mode VPN endpoint virtual machines in much the same wayas described above. The VPN endpoint virtual machine created in responseto the CreateVpnEndpointPeer API request obtains its state for theconnection from the previously created VPN endpoint virtual machine.Such state information may include the security protocols, presharedkey, encryption algorithms, etc.

At this point, the customer has created the VPN endpoint node (which theprovider network has implemented an active/standby mode pair of VPNendpoint virtual machines) as well as the remote node, and the remotenode may be another provider network-hosted VPN endpoint node or acustomer computing device external to the provider network (e.g., agateway). At 316, the method includes establishing a tunnel between theactive mode VPN endpoint node and the remote node and then using thetunnel to exchange packets. As noted above, the tunnel may beimplemented according to the IPSec protocols and the packets exchangedover the IPSec tunnel may be encrypted using IPSec keys generated duringthe tunnel formation process. For example, to implement an IPSec-basedtunnel, the peered VPN endpoint nodes perform a Phase I and a Phase IInegotiation process. During Phase I, the VPN endpoint nodes authenticateeach other using, for example, the preshared key. Each VPN endpoint nodemay compute a hash of its copy of the preshared key (or a set of datathat includes the preshared key) and transmits the resulting hash valueto the peer VPN endpoint node. The receiving VPN endpoint node alsocomputes the hash of its preshared key. Both VPN endpoint nodesauthenticate each other upon determining that the hash values match.Phase I also includes the computation of a Diffie-Hellman key, which isthen used during Phase II to compute an IPSec key. The IPSec key is thekey that is actually used to encrypt packets transmitted across thetunnel. In accordance with the implemented tunnel protocol (e.g.,IPSec), the Phase I (Diffie-Hellman) key and the Phase II key (IPSeckey) may be recomputed from time to time. The frequency with which thekeys are recomputed is configurable and the Phase I key may be computedat a different frequency (e.g., once per day) than the Phase II key(e.g., once per hour). A synchronization process of the keys between theactive mode VPN endpoint virtual machine and its standby modecounterpart helps to ensure that the standby mode VPN endpoint virtualmachine can take over the role of the active mode VPN endpoint virtualmachine if a failure of the active mode VPN endpoint virtual machine isdetected. Examples of the key synchronization process are describedbelow.

FIG. 4 illustrates an embodiment of the system in which the health andstatus of the VPN endpoint virtual machines are monitored. A VPNendpoint node 370 is shown comprising a pair of VPN endpoint virtualmachines 375 a and 375 b. VPN endpoint virtual machine 375 a iscurrently operating in the active mode and VPN endpoint virtual machine37 b is currently operating in the standby mode. A tunnel has beenestablished between the VPN endpoint virtual machine 375 a and a remotepeer node 350. An IP address 380 has been assigned to the VPN endpointvirtual machine 375 a and is used in the packets created and exchangedacross tunnel between the VPN endpoint virtual machine 375 a and theremote peer 350. The IP address 380 is a public IP address that nodesexternal to the provider network use to access the VPN endpoint virtualmachine. The IP address 380 may be an elastic IP address meaning thatthe IP address remains associated with the customer's service provideraccount and, in the event the virtual machine to which the IP address ispresently assigned fails, the IP address can be reassociated withanother virtual machine. IP address 380 may mapped by a network addresstranslator (NAT) to a private IP address within the provider networkwhich is associated with the host computer on which the correspondingVPN endpoint virtual machine is executed. The NAT may maintain a set ofpublic-to-private IP address mappings.

Each VPN endpoint virtual machine 375 a, 375 b sends heartbeat messagesto the health monitoring database 262. The messages may indicate theoperational state of the virtual machine such as explained above. Theheartbeat messages are stored in the health monitoring database for eachof the VPN endpoint virtual machines. The health monitoring service 260can access the heartbeat messages (or lack thereof) of each of the VPNendpoint virtual machines from the database 262 to determine whethereach of the VPN endpoint virtual machines are fully operational or haveexperienced a failure of some type.

FIG. 5 illustrates that the VPN endpoint virtual machine 375 a currentlythe active mode and transacting traffic across the tunnel 360 hasexperienced a failure, as indicated by the “X”. The failure may beencoded in one or more heartbeat messages transmitted from the VPNendpoint virtual machine 375 a and stored in the health monitoringdatabase 262, or the failure may prevent the VPN endpoint virtualmachine 375 a from sending heartbeat messages altogether (or at a ratethat is outside a range of expected heartbeat message updates). Thehealth monitoring service 260 accesses the database and determines basedon the messages or lack thereof that the VPN endpoint virtual machine375 a has experienced a failure. The health monitoring service 260 maycheck the content of the health monitoring database 262 for each VPNendpoint virtual machines at any suitable rate such as once every 10seconds, once per minute, once per hour, etc.

In response to the detection of a failure with the VPN endpoint virtualmachine 375 a currently operating in the active mode, the healthmonitoring service may initiate a fail-over to the currently standbymode VPN endpoint virtual machine 375 b. This process is illustrated inFIGS. 6-9. In FIG. 6, the health monitoring service 390 may send aReAssociateIP API request message to the provisioning service 250. TheIP address 380 may be a public IP address as noted above. TheReAssociateIP API request message may include as input parameters the IPaddress to be reassociated (e.g., IP address 380), an identifier (e.g.,private IP addresses) of the host computer that hosts the failed VPNendpoint virtual machine to which the IP address 380 is currentlyassociated, and the IP address of the host computer that hosts thestandby VPN endpoint virtual machine to which the IP address 380 is tobe re-associated. Address mappings containing IP address 380, such asaddress mappings in a network address translator that maps the public IPaddress to a private address of a provider network host computer, areupdated by the provisioning service 250.

As a result of the IP address mapping updates, the IP address 380 isre-assigned to the VPN endpoint virtual machine 375 b as illustrated inFIG. 7. As a result of re-assigning the IP address 380 for the tunnel360, the tunnel effectively is moved to the VPN endpoint virtual machine375 b. FIG. 7 also illustrates the VPN endpoint virtual machine 375 a ismarked as “unhealthy”. This designation may be implemented by the healthmonitoring service 260 changing a state designation for that virtualmachine in a database, such as the health monitoring database 262.

At this point, the VPN endpoint node, which previously comprised anactive virtual machine and standby virtual machine, now only comprises asingle active virtual machine (virtual machine 375 b, whose modedesignation is shown in FIG. 8 as active instead of standby). The formeractive mode VPN endpoint virtual machine 375 a has experienced a failureand is presumably incapable of adequately functioning to configure andoperate the tunnel 360 to the remote peer 350. Thus, the VPN endpointnode is no longer fault tolerant. However, the health monitoring service260 may submit a RunInstance API call to the provisioning service 250 torequest a new virtual machine to be launched on a host computer tofunction as a standby VPN endpoint virtual machine. FIG. 9 illustratesthe inclusion of a new VPN endpoint virtual machine 397 which has beenlaunched as part of the VPN endpoint node. This virtual machine may belaunched using the same machine image that was used to launch theoriginal pair of virtual machines 375 a and 375 b, and thus may containthe appropriate software capable of performing the functions of a VPNendpoint in the event active mode VPN endpoint 375 b experiences afailure. At this point, the VPN endpoint is again fault tolerant.

FIG. 10 shows a method for failing over from a failed active mode VPNendpoint virtual machine to its standby counterpart in accordance withvarious embodiments. The operations may be performed in the order shown,or in a different order. Further, two or more of the operations may beperformed concurrently instead of sequentially. At 400, the methodincludes the active and standby mode VPN endpoint virtual machinessending heartbeat messages to the health monitoring service 260 asexplained above.

At 402, the health monitoring service 260 determines whether the activemode VPN endpoint virtual machine has failed by, for example, analyzingthe heartbeat message data stored in the health monitoring database 262.If no failure is detected of the active mode VPN endpoint virtualmachine, then control loops back to operation 400 at which the virtualmachines continue to send heart beat messages. The health monitoringservice 260 also may monitor the health and status of the standby modeVPN endpoint virtual machine. Although the standby mode VPN endpointvirtual machine is not presently actively involved in the operation ofthe tunnel to the remote node, it may be desirable to know whether thestandby virtual machine is fully operational. If it is not fullyoperational, the VPN endpoint node is not fault tolerant and correctiveaction may be taken. Such corrective action may include launching areplacement virtual machine to function as the standby VPN endpointvirtual machine.

At any rate, if the active VPN endpoint virtual machine is determined bythe health monitoring service 260 to be experiencing a type of failurethat renders it incapable of adequately functioning as a VPN endpoint,then control continues at operation 404 in which the method includesre-associating, as explained above, the IP address of the failed VPNendpoint virtual machine to the standby mode VPN endpoint virtualmachine. Re-associating the IP address to the former standby mode VPNendpoint virtual machine causes the standby mode VPN endpoint virtualmachine to operate as the active mode VPN endpoint virtual machine.

At 408, the provisioning service, as requested by health monitoringsystem 260, may cause a new virtual machine instance to be launched on ahost computer to replace the failed virtual machine. In someembodiments, the new virtual machine is launched using the same machineimage used to launch the original machine images forming the VPNendpoint node. The newly launched virtual machine may be launched in thesame or different physical or virtual data center as the failed virtualmachine. The newly launched virtual machine is configured to operate inthe standby mode and configuration parameters are transmitted to the newvirtual machine that the instance would need to operate as VPN endpointshould it be needed to operate in the active mode. The configurationparameters may be the parameters discussed previously such as thepre-shared key, the CIDR block addresses, the security protocols, etc.associated with the customer's VPN endpoint, which have been stored aspreviously explained.

At 410, the method also may include updating the mapping between theactive mode VPN endpoint virtual machine and the newly launched standbymode VPN endpoint virtual machine in the key storage service's database.In some embodiments, the newly launched virtual machine subscribesitself with the key storage service to receive any key updates generatedby the newly activated virtual machine by supplying, for example, the IPaddress or other identifying value of the active mode VPN endpointvirtual machine instance. The mappings may include the IDs assigned tothe virtual machines forming the VPN endpoint, the IP addresses of thehost computers on which they operate, and/or any other values thatuniquely identify the particular virtual machines forming the VPNendpoint node. As explained below, the key storage service is used tosynchronize, between the active and standby mode VPN endpoint virtualmachines, the keys (e.g., Phase I Diffie-Hellman key, Phase II IPSeckey, etc.) used to implement the tunnel. The provisioning service 250may transmit one or more messages to the key storage service to updatethe mapping when one of the virtual machines providing fault tolerancefor the VPN endpoint node fails and is replaced with a new virtualmachine.

In some embodiments, the active and standby mode VPN endpoint virtualmachine instances are launched from VPN endpoint-specific machine imagesas noted above. If a software upgrade (e.g., enhanced features, securitypatches, etc.) is made the machine image used to launch VPN endpointvirtual machines, a new pair of virtual machines images can be launchedusing the new machine image and a change-over from the currentlyexecuting active and standby mode VPN endpoint virtual machine instancescan be performed. For example, a pair of new virtual machine instancescan be launched from the newly patched machine image. Once the newvirtual machines have been launched, the provisioning service 250 caninitiate a failover process for each of the current active and standbymode VPN endpoint virtual machine instances to one of the newly launchedvirtual machines. The failover process may include re-associating the IPaddress from the active mode VPN endpoint virtual machine instance toone of the newly launched virtual machine instances that is to operatein the active mode, as well as copying all other relevant stateinformation to the new virtual machines (e.g., encryption keys,encryption protocols, IP address of remote peer, etc.).

FIGS. 11 and 12 illustrate the operation of the key synchronizationprocess. The keys generated in FIGS. 11 and 12 may be keys used toimplement the tunnel between the active mode VPN endpoint virtualmachine of the customer's VPN endpoint node and a remote node (which maybe another VPN endpoint node within the same or different providernetwork or a gateway within the customer's premises. In someembodiments, the tunnel is an IPSec tunnel and thus the keys may theDiffie-Hellman key generated during Phase I or the IPSec key generatedduring Phase II. The Diffie-Hellman and IPSec keys may be generatedperiodically. For example, the Diffie-Hellman keys may be re-computedevery 8 hours, once per day, etc., while the IPSec keys may bere-computed more frequently such as once per hour. In some embodiments,the key synchronization process described in FIGS. 11 and 12 may includesequence numbers that are used during the key regeneration process overthe tunnel.

In FIG. 11, a rekey operation 450 is initiated between the remote peer350 and the active mode VPN endpoint virtual machine 375 a. The rekeyoperation 450 includes performing one or more steps in accordance withthe applicable rekeying protocol. The IPSec protocol includes a seriesof steps for computing Diffie-Hellman keys and a different series ofsteps for computing IPSec keys. Other tunnel-based protocols havedifferent methods for rekey operations. Operation 450 is indicative ofall of the series of steps to perform the applicable rekey operation.For example, some of the steps to compute a Diffie-Hellman key maycomprise each side of the tunnel (remote peer and active mode VPNendpoint virtual machine) to exchange information as to encryptionmethods and algorithms each side supports, compute a private key from apool of random bits, compute a public key from the private key, exchangethe public key with the other side, and compute the Diffie-Hellman keyfrom their own private key and the other side's public key. Some of thesteps to compute an IPSec key may include each side exchanging keymaterial with the other side and establish an agreement on theencryption and integrity methods of IPSec and computing the IPSec keyfrom the Diffie-Hellman key and the key material. As part of the keyregeneration process, the active mode VPN endpoint virtual machine mayreceive a message from the remote peer that includes a sequence numbergenerated by the remote node. The active mode VPN endpoint virtualmachine includes the new sequence number in its message to the keystorage service 280.

After a new key (be it the Diffie-Hellman key, the IPSec key, or anothertype of key) is computed by the active mode VPN endpoint virtual machinebut before the completion of the rekey operation between the remote peer350 and the active mode VPN endpoint virtual machine, operations 452-458are performed to deliver the newly computed key to the standby modevirtual machine. For example, a handshake acknowledgment packet maysignify the end of the rekey operation, and operations 452-458 may beperformed before the active mode VPN endpoint virtual machine 375 atransmits the acknowledgment packet.

At 452, the active mode VPN endpoint virtual machine 375 a sends aKeyPropagate message to the key storage service 280. The KeyPropagatemessage may contain the newly computed key and an identifier of theactive mode VPN endpoint virtual machine 375 a (e.g., the public IPaddress it uses for the tunnel, a virtual machine name, etc.). Theidentifier may be used by the key storage service to update a record inthe key store 282 that corresponds to the identifier. The recordcontains the key(s) used by the active mode VPN endpoint virtual machineand the key storage service may replace the current value of the keywith the newly computed value provided in the KeyPropagate message. Thekeys stored in the key store 282 themselves may be encrypted for addedsecurity. Once the key storage service 282 stores the new value of thekey in key store 282, at 454 the key storage service may send a KeySyncmessage to the standby mode VPN endpoint virtual machine 375 b includingthe new sequence number. The Key Sync message may contain the new valueof the relevant key, which the standby mode VPN endpoint virtual machine375 b receives and stores in its configuration data store. The standbymode VPN endpoint virtual machine 375 b thus now has the key and couldoperate to implement the tunnel should the active mode VPN endpointvirtual machine 375 a experience a failure.

At 456, the standby mode VPN endpoint virtual machine 375 b returns aKeyACK message back (designating the new sequence number) to the keystorage service 280 to indicate that it successfully received the newkey. Once the key storage service 280 receives the KeyACK from thestandby mode VPN endpoint virtual machine 375 b, the key storage service280 transmits a KeyDelivered message at 458 to the active mode VPNendpoint virtual machine 375 a. At this point, the active mode VPNendpoint virtual machine 375 a has received an indication that a copy ofthe newly computed has been received by the standby mode VPN endpointvirtual machine 375 b, and can then complete the rekey operation at 460in accordance with the applicable rekey protocol.

FIG. 12 shows an example of a rekeying operation that is initiated bythe active mode VPN endpoint virtual machine 375 a. The interactionbetween the active and standby mode VPN endpoint virtual machinesthrough the key storage service 280 is the same as described above. At470, the active mode VPN endpoint virtual machine 375 a sends aKeyPropagate message to the key storage service 280. The KeyPropagatemessage contains a newly computed key and an identifier of the activemode VPN endpoint virtual machine 375 a. The active mode VPN maygenerate a new sequence number in this example which propagates throughthe key synchronization messaging as described above. As explainedabove, the identifier may be used by the key storage service to update arecord in the key store 282 that corresponds to the identifier. Therecord contains the key(s) used by the active mode VPN endpoint virtualmachine and the key storage service may replace the current value of thekey with the newly computed value provided in the KeyPropagate message.Once the key storage service 282 stores the new value of the key in keystore 282, at 472 the key storage service sends the KeySync message tothe standby mode VPN endpoint virtual machine 375 b to provide the keyto standby mode VPN endpoint virtual machine. At 474, the standby modeVPN endpoint virtual machine 375 b returns a KeyACK message back to thekey storage service 280 to indicate that it successfully received thenew key, and the key storage service 280 then transmits the KeyDeliveredmessage at 476 to the active mode VPN endpoint virtual machine 375 a. Atthis point, the active mode VPN endpoint virtual machine 375 a and theremote peer 30 can complete the rekey operation at 478 and 480 inaccordance with the applicable rekey protocol.

FIGS. 13A and 14-17 show examples of the use of the fault tolerant VPNendpoint node as described herein, although in some embodiments, the VPNendpoint nodes shown in these examples need not be fault tolerant. Inthis latter case (non-fault tolerant VPN endpoint nodes), a single VPNendpoint virtual machine instance is created to implement thefunctionality of the VPN endpoint to establish a secure tunnel. Theexample of FIG. 13A illustrates two virtual private networks 500 and 520attached to a corresponding VPN endpoint node. VPN endpoint 505 isattached to virtual private network 500, and VPN endpoint 525 isattached to virtual private network 520. Each VPN endpoint 505, 525comprises a plurality of virtual machines, such as two virtual machineseach containing software capable of performing the functionality of aVPN endpoint node and one virtual machine configured to operate in anactive mode and the other virtual machine configured to operate in astandby mode as explained above. A tunnel 530 is established over apublic network 530 such as the Internet between the active mode VPNendpoint virtual machines of each respective VPN endpoint node 505 and525. The two virtual private networks 500, 525 in the example of FIG. 13may be associated with the same customer account or with two differentcustomer accounts. Some service providers may implement multiplephysical and virtual data centers spread across different geographicalregions. Each region may have one or more data centers, but theresources (e.g., computers configured to host virtual machines, virtualmachine provisioning services, storage services, etc.) of the providernetwork in one region may be communicatively coupled to the resources inanother region over public network 530, whereas the resources within aregion may interact with each other via an internal private network ofthat region. As such, a tunnel can be used to establish a secureconnection between virtual private networks in two different regions.

The process of establishing the tunnel 530 is illustrated in FIG. 13B.The operations may be performed in the order shown or in a differentorder. Further, two or more of the operations may be performedconcurrently instead of serially. The various API requests noted belowmay be submitted by a customer and processed by the provisioning service250 or other services within the provider network.

At 531, the method includes creating multiple virtual private networks.In the example of FIG. 13A, virtual private networks 500 and 520 arecreated. This operation may include launching one or more virtualmachines and then assigning virtual machines to a virtual privatenetwork. One or more calls may be submitted to virtual machine creationand virtual private network formation APIs to implement thisfunctionality. As a result of creating a virtual private network, thesystem may assign a unique ID to the virtual private network. As such,one ID is assigned to virtual private network 500, and a separate ID maybe assigned to virtual private network 520.

At 532, the method includes creating a first VPN endpoint node. Thisoperation may include submitting a request to the CreateVpnEndpoint APIto create a new VPN endpoint node such as VPN endpoint node 505 in FIG.13A. The request need not contain any input parameters. As explainedpreviously, the input parameters may include a remote IP address, aremote PSK, a tunnel inside IP CIDR block, a remote BGP AS number, and alocal BGP AS number. If no input parameters are specified, theprovisioning service 250 may generate a PSK string (e.g., a randomstring), provisions a CIDR block of IP addresses (e.g., a /30 CIDR), andgenerates remote and local BGP AS numbers from a private autonomoussystem space (e.g., a value greater than 65,000). The response from theCreateVpnEndpoint API request includes the ID of the newly created VPNendpoint node (e.g., VPN endpoint node 505).

At 533, the process may include attaching the newly created VPN endpointnode (which could be the active mode VPN endpoint virtual machineinstance of a fault tolerant VPN endpoint node) to the correspondingvirtual private network. For example, VPNe 505 may be attached tovirtual private network 500. This operation may include submitting arequest to the AttachVpnEndpoint API. The request may include the ID ofthe first VPN endpoint node created at 532, the ID of the correspondingVPN (e.g., VPN 500), and a subnet ID as applicable. The response fromthe AttachVpnEndpoint API call includes an ID of the VPN endpoint nodeattachment to the VPN. At this point, VPNe 505 is attached to virtualprivate network 500. If the VPN endpoints to be peered together (andtheir respective virtual private networks) are associated with differentcustomer accounts, one customer can provide the other customer with theID of the VPNe of the former customer.

At 534, the process of establishing the tunnel 500 includes creating apeer virtual private network endpoint node using the ID of the first VPNendpoint node created at 532. This operation may include submitting arequest to a CreateVpnEndpointPeer API that includes the ID of the firstVPN endpoint node (e.g., VPNe 505) as an input parameter. Some or all ofthe state information for the first VPN endpoint node (e.g., PSK, tunnelinside CIDR, remote BGP AS number, local BGP AS number, etc.) may beretrieved and used by the VPN endpoint node created at 533.

At 535, the method includes attaching the VPN endpoint created at 533 toits corresponding virtual private network (e.g., attachment of VPNe 525to virtual private network 520). The API for this operation may theAttachVpnEndpoint API as described above, the input parameters for whichmay include the ID of the VPN endpoint created at 533. The ID of therelevant virtual private network and a subnet ID. At this point, datapackets can be encrypted and transmitted across the tunnel between thevirtual private networks (via their respective VPN endpoints).

FIG. 14 shows an example of four virtual private networks 550, 560, 570,and 580. At least one VPN endpoint node is attached to each virtualprivate network, and some virtual private networks have more than oneVPN endpoint node attached thereto. A single VPN endpoint node 555 isattached to virtual private network 550. Two VPN endpoint nodes 563 and565 are attached to virtual private network 560. Two VPN endpoint nodes573 and 575 are attached to virtual private network 570. A single VPNendpoint node 585 is attached to virtual private network 580. As before,the each VPN endpoint may comprise two virtual machines, each containingsoftware capable of performing the functionality of a VPN endpoint nodeand one virtual machine configured to operate in an active mode and theother virtual machine configured to operate in a standby mode.

Between the various VPN endpoint nodes, three tunnels 558, 568, and 578have been established. Tunnel 558 has been established between VPNendpoint nodes 555 and 565 attached respectively to virtual privatenetworks 550 and 560. Similarly, tunnel 568 has been established betweenVPN endpoint nodes 563 and 575 attached respectively to virtual privatenetworks 560 and 570. Finally, tunnel 578 has been established betweenVPN endpoint nodes 573 and 585 attached respectively to virtual privatenetworks 570 and 580. By attaching two VPN endpoint nodes to at leastsome of the virtual private networks in the example of FIG. 14, a meshnetwork is created providing sufficient communication connectivity forinformation to be relayed from any one virtual private network to anyother virtual private network. The creation of the virtual privatenetworks 550, 560, 570, 580, the virtual private network endpoint nodes555, 563, 565, 573, 575, and 585, the tunnels 558, 568, and 578, as wellas the attachment of the virtual private network endpoint nodes to theircorresponding virtual private networks, may be implemented much asdescribed above with respect to the flow diagram of FIG. 13B.

FIG. 15 shows an example similar to that of FIG. 1, but with twoseparate virtual private networks 600 and 610 coupled to a singlegateway 622 in an external (i.e., external to a provider network hostingthe virtual private networks). Virtual provider network 600 is attachedto a VPN endpoint node 605, and virtual provider network 610 is attachedto a VPN endpoint node 615. The gateway 622 may be a component within,for example, a data center of the customer that created the virtualprivate networks 600, 610. Tunnel 612 has been established between VPNendpoint node 605 attached respectively to virtual private network 600and gateway 622. Tunnel 617 has been established between VPN endpointnode 615 attached respectively to virtual private network 610 andgateway 622. The customer that created the virtual private networks 600and 610 can link their various virtual private networks to a commongateway.

Each VPN endpoint 605 and 615 may be created through issuance of arequest to the CreateVpnEndpoint API. The input parameters to the APImay include the IP address of the customer's gateway 622, a PSK, thetunnel inside CIDR, and the remote and local BGP AS numbers as explainedpreviously. Once created, each VPN endpoint node 605 and 615 may beattached its respective virtual private network 600 and 610.

FIG. 16 shows an example of the use of a colocation facility 680. Thecolocation facility includes multiple routers such as routers 682 and684. Router 684 is dedicated for use by a service provider customer andprovides direct dedicated connectivity to a networking device such asgateway 622 in the customer's data center 620. The customer's gateway622 has a direct communication connection to the dedicated router 684.The customer dedicated router 684 can be coupled to other routers withinthe colocation facility such as router 682 which may be owned, operated,and otherwise controlled by the service provider for use by more thanone customer. FIG. 16 shows an example of a customer virtual privatenetwork 650 attached to a fault tolerant VPN endpoint node 655 asdescribed herein. The VPN endpoint node 655 permits the customer toestablish a secure connection (e.g., using encryption) through thecolocation facility to the gateway 622 in the customer's data center620.

In some cases, a customer may want to detach a VPN endpoint from onevirtual private network and attach it to a different virtual privatenetwork. The provisioning service may implement DetachVpnEndpoint andAttachVpnEndpoint APIs for this purpose. The input parameter for theDetachVpnEndpoint API may include the identifier of the currentVPNe-to-virtual private network attachment. In response, theprovisioning service detaches the VPNe from the corresponding virtualprivate network (e.g., updates routing tables or IP address mappings toprevent packets from any of the virtual machines in the virtual privatenetwork from reaching the now detached VPNe node). The input parametersfor the AttachVpnEndpoint API may include the identifier of the VPNenode to be attached to a virtual private network and the identifier ofthe particular virtual private network for the attachment, as well as asubnet identifier. In response, the provisioning service attaches theVPNe node to the specified virtual private network as describedpreviously.

FIG. 17 shows a schematic diagram for a computing system 700 suitablefor implementation of the VPN endpoint virtual nodes and constituentvirtual machines, the virtual machines that comprise virtual privatenetworks, the provisioning service 250, the health monitoring service260, the health monitoring database 262, the mapping service 270, thekey storage service 280 and key store 282, and the various hostcomputers as described herein in accordance with various embodiments.The system includes one or more computing devices 702. The computingsystem 700 includes the computing devices 702 and secondary storage 716communicatively coupled together via a network 718. One or more of thecomputing devices 702 and associated secondary storage 716 may be usedto provide the functionality of the VPN endpoint virtual nodes andconstituent virtual machines, the virtual machines that comprise virtualprivate networks, the provisioning service 250, the health monitoringservice 260, the health monitoring database 262, the mapping service270, and the key storage service 280.

Each computing device 702 includes one or more processors 504 coupled tomemory 506, network interface 712, and I/O devices 714. In someembodiments, a computing device 702 may implement the functionality ofmore than one component of the systems described herein. In variousembodiments, a computing device 702 may be a uniprocessor systemincluding one processor 704, or a multiprocessor system includingseveral processors 704 (e.g., two, four, eight, or another suitablenumber). Processors 504 may be any suitable processor capable ofexecuting instructions. For example, in various embodiments, processors704 may be general-purpose or embedded microprocessors implementing anyof a variety of instruction set architectures (“ISAs”), such as the x86,PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. Inmultiprocessor systems, each of processors 704 may, but not necessarily,commonly implement the same ISA. Similarly, in a distributed computingsystem such as one that collectively implements the provider network110, each of the computing devices 702 may implement the same ISA, orindividual computing nodes and/or replica groups of nodes may implementdifferent ISAs.

The storage 706 may include a non-transitory, computer-readable storagedevice configured to store program instructions 708 and/or data 710accessible by processor(s) 704. The storage 506 may be implemented usingany suitable volatile memory (e.g., random access memory), non-volatilestorage magnetic storage such as a hard disk drive, optical storage,solid storage, etc.). Program instructions 708 and data 710 implementingthe functionality disclosed herein are stored within storage 706. Forexample, instructions 708 may include instructions that when executed byprocessors) 704 implement the VPN endpoint virtual nodes and constituentvirtual machines, the virtual machines that comprise virtual privatenetworks, the provisioning service 250, the health monitoring service260, the health monitoring database 262, the mapping service 270, andkey storage service 280, and/or other components of the serviceprovider's network disclosed herein.

Secondary storage 716 may include additional volatile or non-volatilestorage and storage devices for storing information such as programinstructions and/or data as described herein for implementing thevarious aspects of the service provider's network described herein. Thesecondary storage 716 may include various types of computer-readablemedia accessible by the computing devices 702 via the network 718. Acomputer-readable medium ray include storage media or memory media suchas semiconductor storage, magnetic or optical media, e.g., disk orCD/DVD-ROM, or other storage technologies. Program instructions and datastored on the secondary storage 716 may be transmitted to a computingdevice 502 for execution by a processor 704 by transmission media orsignals via the network 718, which may be a wired or wireless network ora combination thereof. Each of the VPN endpoint virtual nodes andconstituent virtual machines, virtual machines that comprise virtualprivate networks, the provisioning service 250, the health monitoringservice 260, the health monitoring database 262, the mapping service270, and the key storage service 280 and other components describedherein may be implemented as a separate computing device 702 executingsoftware to provide the computing node with the functionality describedherein. In some embodiments, some of the VPN endpoint virtual nodes andconstituent virtual machines, the virtual machines that comprise virtualprivate networks, the provisioning service 250, the health monitoringservice 260, the health monitoring database 262, the mapping service270, and the key storage service 280 and other components may beimplemented by the same computing device.

The network interface 712 may be configured to allow data to beexchanged between computing devices 702 and/or other devices coupled tothe network 718 (such as other computer systems, communication devices,input/output devices, or external storage devices). The networkinterface 712 may support communication via wired or wireless datanetworks, such as any suitable type of Ethernet network, for example;via telecommunications/telephony networks such as analog voice networksor digital fiber communications networks; via storage area networks suchas Fibre Channel SANs, or via any other suitable type of network and/orprotocol.

Input/output devices 714 may include one or more display terminals,keyboards, keypads, touchpads, mice, scanning devices, voice or opticalrecognition devices, or any other devices suitable for entering orretrieving data by one or more computing devices 702. Multipleinput/output devices 714 may be present in a computing device 702 or maybe distributed on various computing devices 702 of the system 700. Insome embodiments, similar input/output devices may be separate fromcomputing device 702 and may interact with one or more computing devices702 of the system 700 through a wired or wireless connection, such asover network interface 712.

References to “based on” should be interpreted as “based at least on.”For example, if a determination of a value or condition is “based on” avalue of Y, then the determination is based at least on the value of Y;the determination may be based on other values as well.

Those skilled in the art will also appreciate that in some embodimentsthe functionality disclosed herein may be provided in alternative ways,such as being split among more software modules or routines orconsolidated into fewer modules or routines. Similarly, in someembodiments illustrated methods may provide more or less functionalitythan is described, such as when other illustrated methods instead lackor include such functionality respectively, or when the amount offunctionality that is provided is altered. In addition, while variousoperations may be illustrated as being performed in a particular manner(e.g., in serial or in parallel) and/or in a particular order, thoseskilled in the art will appreciate that in other embodiments theoperations may be performed in other orders and in other manners. Thevarious methods as depicted in the figures and described hereinrepresent illustrative embodiments of methods. The methods may beimplemented in software, in hardware, or in a combination thereof invarious embodiments. Similarly, the order of any method may be changed,and various elements may be added, reordered, combined, omitted,modified, etc., in various embodiments.

The above discussion is meant to be illustrative of the principles andvarious embodiments of the present disclosure. Numerous variations andmodifications will become apparent to those skilled in the art once theabove disclosure is fully appreciated. It is intended that the followingclaims be interpreted to embrace all such variations and modifications.

What is claimed is:
 1. A system, comprising: one or more computingdevices within a provider network and configured to execute aprovisioning service and a health monitoring service; wherein, inresponse to a virtual private network endpoint creation request to anapplication programming interface (API), the provisioning service isconfigured to launch a virtual private network endpoint (VPNe) virtualmachine on each of two host computers within the provider network;wherein one of the VPNe virtual machines is configured to operate in anactive mode in which that VPNe virtual machine is configured to exchangepackets over an encrypted tunnel; wherein the other one of the VPNevirtual machines is configured to operate in a standby mode in whichthat VPNe virtual machine is configured to affirmatively acknowledge achange in an encryption key used by the active mode VPNe virtualmachine; and wherein the health monitoring service is configured todetermine a health status of the active mode VPNe virtual machine and,upon determination of a failure of the active mode VPNe virtual machine,initiate a fail-over to the standby VPNe virtual machine and cause aninternet protocol (IP) address of the failed active mode VPNe virtualmachine to be associated instead with the standby mode VPNe virtualmachine to continue operation of the encrypted tunnel using the changedencryption key.
 2. The system of claim 1, further comprising a key storeservice containing key storage accessible to, and shared between, theactive and standby mode VPNe virtual machines, the key storageconfigured to store an encryption key usable for the encrypted tunnel,wherein: the active mode VPNe virtual machine is configured to compute areplacement encryption key and to transmit the replacement encryptionkey to the key store service; the key store service is configured storethe replacement key and transmit a key sync message to the standby VPNevirtual machine, the key sync message including a value indicative ofthe replacement key; the standby mode VPNe virtual machine configured totransmit a key acknowledgment message to the key store service; the keystore service transmits a message to the active mode VPNe virtualmachine confirming receipt of the replacement key by the standby modeVPNe virtual machine; and in response to the message from the key storeservice confirming receipt of the replacement key by the standby modeVPNe virtual machine, the active mode VPNe virtual machine completes arekey operation of the replacement key over the encrypted tunnel.
 3. Thesystem of claim 1, wherein: the active mode VPNe virtual machine isconfigured to establish a communication tunnel using state informationprovided by the provisioning service; the health monitoring service isconfigured to send a message to the provisioning service indicating anoccurrence of a failure of the active mode VPNe virtual machine; and inresponse to the message, the provisioning service causes a new virtualmachine to be launched on a host computer and the state information tobe provided to the new virtual machine.
 4. The system of claim 1,wherein the provisioning service is configured to cause configurationparameters to be transmitted to the host computers for use within theactive and standby mode VPNe virtual machines, the configurationparameters including at least one of: an IP address of a remotecomputing node attached to the encrypted tunnel, a pre-shared key, aclassless inter-domain routing (CIDR) IP address block, and a BorderGateway Protocol (BGP) autonomous system (AS) number.
 5. The system ofclaim 1, wherein each of the active mode and standby mode VPNe virtualmachines is configured to periodically transmit a heartbeat message to aheath monitoring database, the heartbeat message indicating a healthstatus of the respective VPNe virtual machine, and wherein the healthmonitoring service is configured to determine the health status of theactive and standby mode VPNe virtual machines through access of thehealth monitoring database.
 6. A system, comprising: one or morecomputing devices within a provider network and configured to execute aprovisioning service; wherein, in response to a virtual private networkendpoint creation request to an application programming interface (API),the provisioning service is configured to launch a first virtual privatenetwork endpoint (VPNe) virtual machine and a second VPNe virtualmachine on one or more server computers within a provider network;wherein the first VPNe virtual machine is configured to implement acommunication tunnel for packet exchange using state informationprovided by the provisioning service to configure the communicationtunnel; and wherein the second VPNe virtual machine is configured tooperate as a standby VPNe virtual machine, the second VPNe virtualmachine includes the state information used by the first VPNe virtualmachine to configure the communication tunnel, and the second VPNevirtual machine is configured to affirmatively receive and acknowledge achange in a key generated by the first VPNe virtual machine.
 7. Thesystem of claim 6, wherein: the one or more computing devices within theprovider network are configured to execute a health monitoring service;each of the first and second VPNe virtual machines is configured totransmit health and status messages to a database; the health monitoringservice is configured to access the database and to determine anoccurrence of an error with the first VPNe virtual machine and totransmit a message to the provisioning service indicative of the errorof the first VPNe virtual machine; and in response to the message fromthe health monitoring service, the provisioning service initiates areassignment of an internet protocol (IP) address from the first VPNevirtual machine to the second VPNe virtual machine.
 8. The system ofclaim 7, wherein the provisioning service is configured to transmitconfiguration data to the second VPNe virtual machine containing thereassigned IP address and the state information, and the second VPNevirtual machine uses the reassigned IP address, the changed key, and thestate information to send and receive encrypted traffic across thetunnel.
 9. The system of claim 8, wherein in response to the messagefrom the health monitoring service, the provisioning service isconfigured to launch a third VPNe virtual machine on the one or moreserver computers within the provider network and to provide the thirdVPNe virtual machine with the state information used by the second VPNevirtual machine to send and received encrypted traffic across thetunnel; wherein the third VPNe virtual machine is configured toaffirmatively receive and acknowledge a change in a key generated by thesecond VPNe virtual machine.
 10. The system of claim 6, wherein thecommunication tunnel is an encrypted communication tunnel.
 11. Thesystem of claim 6, wherein the key includes a Diffie-Hellman key or anInternet Protocol Security (IPSec key).
 12. The system of claim 6,wherein the first VPNe virtual machine is configured to participate in arekey operation to change the key, and wherein, before completion of therekey operation, the first VPNe virtual machine is configured totransmit a message containing a newly computed key, and the second VPNevirtual machine is configured to receive the newly computed key and togenerate an acknowledgement message indicating successful receipt of thenewly computed key.
 13. The system of claim 12, wherein the first VPNevirtual machine is configured to complete the rekey operation across thecommunication tunnel after receipt of a message indicating thesuccessful receipt of the newly computed key by the second VPNe virtualmachine.
 14. The system of claim 6, further comprising a key storeservice containing key storage accessible to, and shared between, thefirst and second VPNe virtual machines, the key storage configured tostore the key usable for the communication tunnel, and wherein: thefirst VPNe virtual machine is configured compute a replacementencryption key and to transmit the replacement encryption key to the keystore service; the key store service is configured store the replacementkey and transmit a key sync message to the second VPNe virtual machine,the key sync message including a value indicative of the replacementkey; the second VPNe virtual machine is configured to transmit a keyacknowledgment message to the key store service; the key store serviceis configured to transmit a message to the first VPNe virtual machineconfirming receipt of the replacement key by the second VPNe virtualmachine.
 15. A method, comprising: in response to a request to create avirtual private network endpoint (VPNe) node, launching a first andsecond VPNe virtual machines on one or more server computers, each ofthe first and second VPNe virtual machines launched using a separatecopy of a machine image that includes a software application that isconfigured to cause the corresponding VPNe virtual machine to configurea communication tunnel and send and receive encrypted packets over thecommunication tunnel; transmitting state information to each of thefirst and second VPNe virtual machines, the state information usable toconfigure a communication tunnel; configuring, by the first VPNe virtualmachine, the communication tunnel using the state information andencrypting packets using a first encryption key to be transmitted acrossthe communication tunnel; during a rekeying operation using thecommunication tunnel, computing a second encryption key, by the firstVPNe virtual machine; synchronizing the second encryption key to thesecond VPNe virtual machine; and completing the rekeying operation afterthe synchronizing.
 16. The method of claim 15, wherein synchronizing thesecond encryption key to the second VPNe virtual machine comprisestransmitting the second encryption from the first VPNe virtual machineto the second VPNe virtual machine through a key storage service whichis configured to store a copy of the second encryption key.
 17. Themethod of claim 15, wherein synchronizing the second encryption key tothe second VPNe virtual machine comprises transmitting a keyacknowledgment message from the second VPNe virtual machineacknowledging receipt by the second VPNe virtual machine of the secondencryption key.
 18. The method of claim 15, further comprising: using aninternet protocol (IP) address by the first VPNe virtual machine totransmit encrypted packets over the communication tunnel to adestination node; detecting a failure of the first VPNe virtual machine;transmitting the IP address used by the first VPNe virtual machine tothe second VPNe virtual machine; and using the IP address by the secondVPNe virtual machine, transmitting encrypted packets over thecommunication tunnel to the destination node.
 19. The method of claim18, further comprising: launching a third VPNe virtual machine on theone or more server computers using a separate copy of the same machineimage used to launch the first and second VPNe virtual machines;synchronizing the first encryption key to the third VPNe virtual machinethrough a key storage service.
 20. The method of claim 18, whereindetecting the failure of the first VPNe virtual machine comprisesanalyzing a health monitoring database configured to include heartbeatmessages from the first VPNe virtual machine.